A reaction norm perspective on reproducibility

Reproducibility in biomedical research, and more specifically in preclinical animal research, has been seriously questioned. Several cases of spectacular failures to replicate findings published in the primary scientific literature have led to a perceived reproducibility crisis. Diverse threats to reproducibility have been proposed, including lack of scientific rigour, low statistical power, publication bias, analytical flexibility and fraud. An important aspect that is generally overlooked is the lack of external validity caused by rigorous standardization of both the animals and the environment. Here, we argue that a reaction norm approach to phenotypic variation, acknowledging gene-by-environment interactions, can help us seeing reproducibility of animal experiments in a new light. We illustrate how dominating environmental effects can affect inference and effect size estimates of studies and how elimination of dominant factors through standardization affects the nature of the expected phenotype variation through the reaction norms of small effect. Finally, we discuss the consequences of reaction norms of small effect for statistical analysis, specifically for random effect latent variable models and the random lab model

Behavioral Profiles of Genetically Selected Aggressive and Nonaggressive Male Wild House Mice in Two Anxiety Tests

Artificially selected aggressive (SAL) and non-aggressive (LAL) male house mice were tested in a hexagonal tunnel maze and light–dark preference (LD) box to determine if the bidirectional selection for aggressive behavior leads to a coselection for different levels of trait anxiety. The tunnel maze consists of an open, brightly lit central arena surrounded by a complex system of interconnecting tunnels. As in the LD box, animals which spend less time and are less active in the brightly illuminated section of the maze are considered to have higher anxiety levels. In the tunnel maze, the LAL mice showed more exploration and spent more time in the central arena than the SAL animals, but only during the final 2 min of the 6-min test. This reduced preference for the central arena was not due to general inactivity or a failure of the SAL to find the central arena and indicates a higher level of state anxiety in the aggressive animals. In contrast, no “anxiety-like” differences were found in the LD box, either for the percentage of time spent in the light compartment or for the number of crossings. SAL males actually showed higher levels of moving and rearing, and lower levels of freezing, than did LAL males.

Handling method affects measures of anxiety, but not chronic stress in mice.

Studies in mice have shown that less aversive handling methods (e.g. tunnel or cup handling) can reduce behavioural measures of anxiety in comparison to picking mice up by their tail. Despite such evidence, tail handling continues to be used routinely. Besides resistance to change accustomed procedures, this may also be due to the fact that current evidence in support of less aversive handling is mostly restricted to effects of extensive daily handling, which may not apply to routine husbandry practices. The aim of our study was to assess whether, and to what extent, different handling methods during routine husbandry induce differences in behavioural and physiological measures of stress in laboratory mice. To put the effects of handling method in perspective with chronic stress, we compared handling methods to a validated paradigm of unpredictable chronic mild stress (UCMS). We housed mice of two strains (Balb/c and C57BL/6) and both sexes either under standard laboratory conditions (CTRL) or under UCMS. Half of the animals from each housing condition were tail handled and half were tunnel handled twice per week, once during a cage change and once for a routine health check. We found strain dependent effects of handling method on behavioural measures of anxiety: tunnel handled Balb/c mice interacted with the handler more than tail handled conspecifics, and tunnel handled CTRL mice showed increased open arm exploration in the elevated plus-maze. Mice undergoing UCMS showed increased plasma corticosterone levels and reduced sucrose preference. However, we found no effect of handling method on these stress-associated measures. Our results therefore indicate that routine tail handling can affect behavioural measures of anxiety, but may not be a significant source of chronic husbandry stress. Our results also highlight strain dependent responses to handling methods

Development of a data-driven method for assessing health and welfare in the most common livestock species in Switzerland: The Smart Animal Health project.

Improving animal health and welfare in livestock systems depends on reliable proxies for assessment and monitoring. The aim of this project was to develop a novel method that relies on animal-based indicators and data-driven metrics for assessing health and welfare at farm level for the most common livestock species in Switzerland. Method development followed a uniform multi-stage process for each species. Scientific literature was systematically reviewed to identify potential health and welfare indicators for cattle, sheep, goats, pigs and poultry. Suitable indicators were applied in the field and compared with outcomes of the Welfare Quality® scores of a given farm. To identify farms at risk for violations of animal welfare regulations, several agricultural and animal health databases were interconnected and various supervised machine-learning techniques were applied to model the status of farms. Literature reviews identified a variety of indicators, some of which are well established, while others lack reliability or practicability, or still need further validation. Data quality and availability strongly varied among animal species, with most data available for dairy cows and pigs. Data-based indicators were almost exclusively limited to the categories "Animal health" and "Husbandry and feeding". The assessment of "Appropriate behavior" and "Freedom from pain, suffering, harm and anxiety" depended largely on indicators that had to be assessed and monitored on-farm. The different machine-learning techniques used to identify farms for risk-based animal welfare inspections reached similar classification performances with sensitivities above 80%. Features with the highest predictive weights were: Participation in federal ecological and animal welfare programs, farm demographics and farmers' notification discipline for animal movements. A common method with individual sets of indicators for each species was developed. The results show that, depending on data availability for the individual animal categories, models based on proxy data can achieve high correlations with animal health and welfare assessed on-farm. Nevertheless, for sufficient validity, a combination of data-based indicators and on-farm assessments is currently required. For a broad implementation of the methods, alternatives to extensive manual on-farm assessments are needed, whereby smart farming technologies have great potential to support the assessment if the specific monitoring goals are defined

Reliability of common mouse behavioural tests of anxiety: A systematic review and meta-analysis on the effects of anxiolytics.

The validity of widely used rodent behavioural tests of anxiety has been questioned, as they often fail to produce consistent results across independent replicate studies. In this study, we assessed the sensitivity of common behavioural tests of anxiety in mice to detect anxiolytic effects of drugs prescribed to treat anxiety in humans. We conducted a pre-registered systematic review of 814 studies reporting effects of 25 anxiolytic compounds using common behavioural tests for anxiety. Meta-analyses of effect sizes of treatments showed that only two out of 17 commonly used test measures reliably detected effects of anxiolytic compounds. We report considerable between-study variation in size and even direction of effects of most anxiolytics on most outcome variables. Our findings indicate a general lack of sensitivity of those behavioural tests and cast serious doubt on both construct and predictive validity of most of these tests. In view of scientifically valid and ethically responsible research, we call for a revision of behavioural tests of anxiety in mice and the development of more predictive tests

Do multiple experimenters improve the reproducibility of animal studies?

The credibility of scientific research has been seriously questioned by the widely claimed "reproducibility crisis". In light of this crisis, there is a growing awareness that the rigorous standardisation of experimental conditions may contribute to poor reproducibility of animal studies. Instead, systematic heterogenisation has been proposed as a tool to enhance reproducibility, but a real-life test across multiple independent laboratories is still pending. The aim of this study was therefore to test whether heterogenisation of experimental conditions by using multiple experimenters improves the reproducibility of research findings compared to standardised conditions with only one experimenter. To this end, we replicated the same animal experiment in 3 independent laboratories, each employing both a heterogenised and a standardised design. Whereas in the standardised design, all animals were tested by a single experimenter; in the heterogenised design, 3 different experimenters were involved in testing the animals. In contrast to our expectation, the inclusion of multiple experimenters in the heterogenised design did not improve the reproducibility of the results across the 3 laboratories. Interestingly, however, a variance component analysis indicated that the variation introduced by the different experimenters was not as high as the variation introduced by the laboratories, probably explaining why this heterogenisation strategy did not bring the anticipated success. Even more interestingly, for the majority of outcome measures, the remaining residual variation was identified as an important source of variance accounting for 41% (CI95 [34%, 49%]) to 72% (CI95 [58%, 88%]) of the observed total variance. Despite some uncertainty surrounding the estimated numbers, these findings argue for systematically including biological variation rather than eliminating it in animal studies and call for future research on effective improvement strategies

The rearing environment persistently modulates mouse phenotypes from the molecular to the behavioural level.

The phenotype of an organism results from its genotype and the influence of the environment throughout development. Even when using animals of the same genotype, independent studies may test animals of different phenotypes, resulting in poor replicability due to genotype-by-environment interactions. Thus, genetically defined strains of mice may respond differently to experimental treatments depending on their rearing environment. However, the extent of such phenotypic plasticity and its implications for the replicability of research findings have remained unknown. Here, we examined the extent to which common environmental differences between animal facilities modulate the phenotype of genetically homogeneous (inbred) mice. We conducted a comprehensive multicentre study, whereby inbred C57BL/6J mice from a single breeding cohort were allocated to and reared in 5 different animal facilities throughout early life and adolescence, before being transported to a single test laboratory. We found persistent effects of the rearing facility on the composition and heterogeneity of the gut microbial community. These effects were paralleled by persistent differences in body weight and in the behavioural phenotype of the mice. Furthermore, we show that environmental variation among animal facilities is strong enough to influence epigenetic patterns in neurons at the level of chromatin organisation. We detected changes in chromatin organisation in the regulatory regions of genes involved in nucleosome assembly, neuronal differentiation, synaptic plasticity, and regulation of behaviour. Our findings demonstrate that common environmental differences between animal facilities may produce facility-specific phenotypes, from the molecular to the behavioural level. Furthermore, they highlight an important limitation of inferences from single-laboratory studies and thus argue that study designs should take environmental background into account to increase the robustness and replicability of findings

Report of the Working Group on Animal Distress in the Laboratory

Finding ways to minimize pain and distress in research animals is a continuing goal in the laboratory animal research field. Pain and distress, however, are not synonymous, and often measures that alleviate one do not affect the other. Here, the authors provide a summary of a meeting held in February 2004 that focused on distress in laboratory animals. They discuss the difficulties associated with defining ‘distress,’ propose methods to aid in recognizing and alleviating distressful conditions, and provide recommendations for animal research conduct and oversight that would minimize distress experienced by laboratory animals

Systematic assessment of the replicability and generalizability of preclinical findings: Impact of protocol harmonization across laboratory sites.

The influence of protocol standardization between laboratories on their replicability of preclinical results has not been addressed in a systematic way. While standardization is considered good research practice as a means to control for undesired external noise (i.e., highly variable results), some reports suggest that standardized protocols may lead to idiosyncratic results, thus undermining replicability. Through the EQIPD consortium, a multi-lab collaboration between academic and industry partners, we aimed to elucidate parameters that impact the replicability of preclinical animal studies. To this end, 3 experimental protocols were implemented across 7 laboratories. The replicability of results was determined using the distance travelled in an open field after administration of pharmacological compounds known to modulate locomotor activity (MK-801, diazepam, and clozapine) in C57BL/6 mice as a worked example. The goal was to determine whether harmonization of study protocols across laboratories improves the replicability of the results and whether replicability can be further improved by systematic variation (heterogenization) of 2 environmental factors (time of testing and light intensity during testing) within laboratories. Protocols were tested in 3 consecutive stages and differed in the extent of harmonization across laboratories and standardization within laboratories: stage 1, minimally aligned across sites (local protocol); stage 2, fully aligned across sites (harmonized protocol) with and without systematic variation (standardized and heterogenized cohort); and stage 3, fully aligned across sites (standardized protocol) with a different compound. All protocols resulted in consistent treatment effects across laboratories, which were also replicated within laboratories across the different stages. Harmonization of protocols across laboratories reduced between-lab variability substantially compared to each lab using their local protocol. In contrast, the environmental factors chosen to introduce systematic variation within laboratories did not affect the behavioral outcome. Therefore, heterogenization did not reduce between-lab variability further compared to the harmonization of the standardized protocol. Altogether, these findings demonstrate that subtle variations between lab-specific study protocols may introduce variation across independent replicate studies even after protocol harmonization and that systematic heterogenization of environmental factors may not be sufficient to account for such between-lab variation. Differences in replicability of results within and between laboratories highlight the ubiquity of study-specific variation due to between-lab variability, the importance of transparent and fine-grained reporting of methodologies and research protocols, and the importance of independent study replication